Automatic extraction of assets data from engineering data sources

ABSTRACT

Systems and methods for controlling industrial an industrial plant comprise: inputting an engineering diagram for a unit of the industrial plant, the engineering diagram including symbols representing assets of the industrial plant; extracting one or more assets from the engineering diagram using machine learning to recognize the one or more assets, the one or more assets including equipment, instruments, connectors, and lines, the lines relating the equipment, instruments, and connectors to one another; determining one or more relationships between the equipment, instruments, connectors, and lines to one another using machine learning to recognize the one or more relationships; and creating a flow graph from the equipment, instruments, connectors, and lines and the relationships between the equipment, instruments, connectors, and lines.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application for patent claims the benefit of priority to andincorporates herein by reference U.S. Provisional Application No.62/823,469, entitled “Systems and Methods for Performing IndustrialPlant Diagnostics and Operations,” filed Mar. 25, 2019; U.S. ProvisionalApplication No. 62/842,929, entitled “Systems and Methods for PerformingIndustrial Plant Diagnostics,” filed May 3, 2019; and U.S. ProvisionalApplication No. 62/823,377, entitled “Systems and Methods for Detectingand Predicting Faults in an Industrial Process Automation System,” filedMar. 25, 2019. This application is also related in subject matter toU.S. Non-Provisional application Ser. No. 16/021,867, entitled “MachineLearning Analysis of Piping and Instrumentation Diagrams,” filed Jun.28, 2018, which application is incorporated herein by reference.

TECHNICAL FIELD

Aspects of the present disclosure generally relate to industrial processautomation and control systems. More particularly, aspects of thepresent disclosure relate to systems and method for extracting plantassets from engineering diagrams and other engineering data sources,creating a plant asset hierarchy from the plant assets, building anontological knowledge base from the plant asset hierarchy and the assetassociation relationships with other assets, and providing ahuman-machine interface (“HMI”) based on the plant asset hierarchy andthe ontological knowledge base for controlling industrial processautomation and control systems.

BACKGROUND

Plant operators are tasked with ensuring an industrial plant operatesproperly. This entails, for example, monitoring for and addressingalarms triggered by plant components (e.g., pumps, valves, tanks,sensors), performing operations on plant components (e.g., shutting downcomponents and starting up components), and generally overseeing properoperation of the plant. To perform these tasks, plant operators may usean HMI that provides plant operators with a visual representation ofplant components and data collected from plant components. The HMI mayallow plant operators to interact with plant components (e.g., toperform an operation such as shutting down a component).

An HMI may be designed for a particular plant to capture the componentswithin the plant, display data used to operate the plant, and provide aninterface for initiating operations that may be desired by plantoperators. Designing the HMI may be costly and time-consuming, and mayrequire specialized knowledge of plant and process engineering. Today,plant HMIs are created using a process-centric approach that focuses onprocess definitions. A plant operator, on the other hand, has a moreasset-centric view of the plant that focuses on automation controldefinitions (e.g., plant, area, unit, equipment, devices, set points ofspecific equipment, etc.). Thus, plant operators may find current plantHMIs to be non-intuitive and difficult to use.

Accordingly, improvements are needed in the field of industrial plantcommissioning and operations.

SUMMARY

Embodiments of the present disclosure provide systems and methods forcontrolling industrial process automation and control systems. Themethods and systems automatically, and through the use of machinelearning (ML) models and algorithms, extract plant assets fromengineering diagrams and other plant engineering data sources, establishasset relationships to create a plant asset registry and build an assethierarchy from the plant assets, generate an ontological knowledge basefrom the plant asset hierarchy, and provide an HMI for controlling theindustrial process based on the plant asset hierarchy and theontological knowledge base.

In general, in one aspect, embodiments of the present disclosure relateto a control system for an industrial plant and corresponding methodtherefor. The control system (and method therefor) comprises, amongother things, one or more processors and a storage unit communicativelycoupled to the one or more processors. The storage unit storesprocessor-executable instructions that, when executed by the one or moreprocessors, cause the control system to input an engineering diagram fora unit of the industrial plant, the engineering diagram includingsymbols representing assets of the industrial plant, and extract one ormore assets from the engineering diagram using machine learning torecognize the one or more assets. The one or more assets includeequipment, instruments, connectors, and lines, the lines relating theequipment, instruments, and connectors to one another. Theprocessor-executable instructions also cause the control systemdetermine one or more relationships between the equipment, instruments,connectors, and lines to one another using machine learning to recognizethe one or more relationships. The processor-executable instructionsalso cause the control system to create a flow graph from the equipment,instruments, connectors, and lines and the relationships between theequipment, instruments, connectors, and lines.

In accordance with any one or more of the foregoing embodiments, theprocess that extracts one or more assets from the engineering diagramincludes a geometry-based extraction process and a machinelearning-based classification process. In accordance with any one ormore of the foregoing embodiments, the processor-executable instructionsfurther cause the control system to extract one or more control loopsfrom the engineering diagram based on one or more of the equipment,instruments, connectors, and lines. In accordance with any one or moreof the foregoing embodiments, the processor-executable instructionsfurther cause the control system to extract a unit identifier from theengineering diagram, wherein the unit identifier uniquely identifies aparticular unit from among multiple units in the industrial plant,and/or extract a drawing number and a revision number from theengineering diagram, the drawing number uniquely identifying theengineering diagram from among multiple engineering diagrams for theindustrial plant and the revision number indicating a revision of theengineering diagram which is incremented whenever there are changes inthe engineering diagram. In accordance with any one or more of theforegoing embodiments, the processor-executable instructions furthercause the control system to assign a unique identifier to eachequipment, instrument, connector, and line of the engineering diagramusing the unit identifier from the engineering diagram. In accordancewith any one or more of the foregoing embodiments, the process thatdetermines one or more relationships between the equipment, instruments,connectors, and lines includes a process that generates a line-linegraph, a process that generates an equipment-line graph, a process thatgenerates a connector-line graph, and a process that generates aninstrument-line graph. In accordance with any one or more of theforegoing embodiments, the processor-executable instructions furthercause the control system to create the flow graph for the engineeringdiagram by performing a process that merges the line-line graph, theequipment-line graph, the connector-line graph, and the instrument-linegraph with one another, and/or merge flow graphs from multipleengineering diagrams for the industrial plant to create a flow graph forthe industrial plant. In accordance with any one or more of theforegoing embodiments, the processor-executable instructions furthercause the control system to display the equipment, instruments,connectors and lines and the relationships between the equipment,instruments, connectors and lines in an asset hierarchy.

In general, in another aspect, embodiments of the present disclosurerelate to a control system for an industrial plant and correspondingmethod therefor. The control system (and method therefor) comprises,among other things, one or more processors and a storage unitcommunicatively coupled to the one or more processors. The storage unitstores processor-executable instructions that, when executed by the oneor more processors, cause the control system to input data from aplurality of engineering data sources for the industrial plant, the dataincluding structured and unstructured data, and extract one or moredomain entities from the structured and unstructured data. Theprocessor-executable instructions also cause the control system toextract instances of the one or more domain entities from the structuredand unstructured data, and receive a semantic model built based on theone or more domain entities and the instances of one or more domainentities. The processor-executable instructions further cause thecontrol system to create and store a knowledge graph based on thesemantic model, and extract information from the knowledge graph tobuild machine interface (HMI) displays and control applications for theindustrial plant.

In accordance with any one or more of the foregoing embodiments, theprocessor-executable instructions further cause the control system toextract one or more assets of the industrial plant from the unstructureddata using machine learning to recognize the one or more assets, and/orstore the one or more assets of the industrial plant in an assethierarchy using a namespace that uniquely identifies each asset. Inaccordance with any one or more of the foregoing embodiments, theprocessor-executable instructions further cause the control system toprovide an asset registry application programming interface (API) forallowing users and external systems to access and use the assetregistry. In accordance with any one or more of the foregoingembodiments, the processor-executable instructions further cause thecontrol system to input data from the plurality of engineering datasources by inputting one or more of: alarm data, cabinet loading report,control database, cross wiring report, field wiring index, historicaldata, instrument index, asset hierarchy, nest loading report, plantasset index, process control narrative, and HMI specification. Inaccordance with any one or more of the foregoing embodiments, theprocessor-executable instructions further cause the control system todisplay the knowledge base as a plurality of interlinkedlabeled-property nodes, each node representing a domain entity, and/ordisplay properties and relationships for each node when the node isselected by a user. In accordance with any one or more of the foregoingembodiments, the processor-executable instructions further cause thecontrol system to allow a user to conduct a search of the knowledge baseusing natural language queries, display nodes responsive to the searchalong with assets that the nodes share in common, and/or display alegend for the nodes on a color-coded basis.

In general, in yet another aspect, embodiments of the present disclosurerelate to a control system for an industrial plant and correspondingmethod therefor. The control system (and method therefor) comprises,among other things, one or more processors and a storage unitcommunicatively coupled to the one or more processors. The storage unitstores processor-executable instructions that, when executed by the oneor more processors, cause the control system to input data from aplurality of engineering data sources for the industrial plant, andextract one or more assets of the industrial plant from the data usingmachine learning to identify the one or more assets. Theprocessor-executable instructions also cause the control system tocreate an HMI (human-machine interface) asset model from the assets, theHMI asset model arranging the assets in a hierarchical structure, andgenerate an HMI display from the HMI asset model, the HMI displaydisplaying symbols that represent the one or more assets of theindustrial plant and lines that represent processes, connections, anddata links between the assets. The processor-executable instructionsalso cause the control system to display the HMI display to a user andallows the user to navigate vertically and horizontally along thedisplay, wherein the HMI display dynamically changes which assets aredisplayed based on a change in position of the user on the HMI display.

In accordance with any one or more of the foregoing embodiments, theprocessor-executable instructions further cause the control system toidentify all alarms currently raised for the industrial plant and assetscorresponding to the alarms and displays the alarms on the HMI display,and/or display the HMI display at a plant level, the plant leveldisplaying all currently raised alarms identified for the industrialplant and assets corresponding to the alarms. In accordance with any oneor more of the foregoing embodiments, the processor-executableinstructions further cause the control system to determine and displayon the HMI display assets that are potential root causes for an alarmupon the user selecting an asset corresponding to one of the alarms,wherein the control system determines the assets that are potential rootcauses by finding all assets that are connected to one another that alsohave an alarm, and/or allows a user to manually correct a root cause ofan alarm on the HMI display, and provides the root cause from the useras feedback to the control system. In accordance with any one or more ofthe foregoing embodiments, the processor-executable instructions furthercause the control system to automatically zoom in or zoom out on theassets displayed on the HMI display based on a change in a position ofthe user on the HMI screen. In accordance with any one or more of theforegoing embodiments, the processor-executable instructions furthercause the control system to display assets in the HMI display as atwo-dimensional view, wherein the HMI display allows a user to navigatevertically and horizontally within the two-dimensional view. Inaccordance with any one or more of the foregoing embodiments, theprocessor-executable instructions further cause the control system todynamically change the assets displayed on HMI display based on an alarmoccurring on an asset, and or display assets on the HMI display atruntime based on one of: static weight assigned to an asset based on aposition of the asset in an asset hierarchy, and dynamic weight assignedto the asset based on an alarm being raised at the asset.

In general, in still another aspect, embodiments of the presentdisclosure relate to a computer-readable medium storingcomputer-readable instruction for causing one or more processors toperform a method according to any one more of the foregoing embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

A more detailed description of the disclosure, briefly summarized above,may be had by reference to various embodiments, some of which areillustrated in the appended drawings. While the appended drawingsillustrate select embodiments of this disclosure, these drawings are notto be considered limiting of its scope, for the disclosure may admit toother equally effective embodiments.

FIG. 1 illustrates an exemplary plant asset hierarchy according toembodiments of the present disclosure;

FIG. 2 illustrates an exemplary engineering diagram that may be used tobuild a plant asset hierarchy according to embodiments of the presentdisclosure;

FIG. 3 illustrates an exemplary industrial plant control system that maybe used to process according to embodiments of the present disclosure;

FIG. 4 illustrate an exemplary intelligent processing module that may beused with the industrial plant control system according to embodimentsof the present disclosure;

FIG. 5 illustrates an exemplary HMI application that may be used withthe industrial plant control system according to embodiments of thepresent disclosure;

FIG. 6 illustrates exemplary ML-based asset extraction and relationshipbuilding according to embodiments of the present disclosure;

FIGS. 7A-7B illustrate an exemplary arrow extraction according toembodiments of the present disclosure;

FIG. 8 illustrates an exemplary pump extraction according to embodimentsof the present disclosure;

FIGS. 9 and 9A-9C illustrates an exemplary connector extractionaccording to embodiments of the present disclosure;

FIGS. 10A-10B illustrates an exemplary line-line graph according toembodiments of the disclosure;

FIG. 11 illustrates an exemplary equipment extraction according toembodiments of the present disclosure;

FIGS. 12A-12B illustrate an exemplary equipment-line graph according toembodiments of the disclosure;

FIGS. 13A-13B illustrate an exemplary connector-line graph according toembodiments of the disclosure;

FIGS. 14A-14B illustrate an exemplary instrument-line graph according toembodiments of the disclosure;

FIGS. 15A-16B illustrate an PI&D flow graph according to embodiments ofthe disclosure;

FIGS. 16A-15B illustrate an exemplary line-line graph with devicesremoved according to embodiments of the disclosure;

FIG. 17 illustrates an exemplary control loop according to embodimentsof the disclosure;

FIG. 18 illustrates an exemplary method for building a plant engineeringdomain ontology according to embodiments of the disclosure;

FIG. 19 illustrates exemplary alarm data according to embodiments of thedisclosure;

FIG. 20 illustrates an exemplary cabinet loading report according toembodiments of the disclosure;

FIG. 21 illustrates an exemplary control database according toembodiments of the disclosure;

FIG. 22 illustrates an exemplary cross wiring report according toembodiments of the disclosure;

FIG. 23 illustrates an exemplary field wiring index according toembodiments of the disclosure;

FIG. 24 illustrates exemplary historical data according to embodimentsof the disclosure;

FIG. 25 illustrates an exemplary instrument index according toembodiments of the disclosure;

FIG. 26 illustrates an exemplary asset hierarchy according toembodiments of the disclosure;

FIG. 27 illustrates an exemplary nest loading report according toembodiments of the disclosure;

FIG. 28 illustrates an exemplary plant asset index according toembodiments of the disclosure;

FIG. 29 illustrates an exemplary process control narrative according toembodiments of the disclosure;

FIG. 30 illustrates an exemplary tabular format for data extractionaccording to embodiments of the disclosure;

FIG. 31 illustrates an exemplary user interface for building a semanticmodel using entity types according to embodiments of the disclosure;

FIG. 32 illustrates an exemplary HMI display screen showing a knowledgegraph according to embodiments of the disclosure;

FIG. 33 illustrates an exemplary HMI display screen showing user queryresults according to embodiments of the disclosure;

FIG. 34 illustrates an exemplary HMI design specification according toembodiments of the disclosure;

FIG. 35 illustrates exemplary inputs for building an HMI model accordingto embodiments of the disclosure;

FIG. 36 illustrates exemplary HMI screen showing alarm root causedetermination according to embodiments of the disclosure;

FIG. 37 illustrates an exemplary HMI screen showing plant level alarmaggregation according to embodiments of the disclosure;

FIG. 38 illustrates exemplary dynamic zooming according to user positionon an HMI screen according to embodiments of the disclosure;

FIG. 39 illustrates exemplary HMI screens showing dynamic zoomingaccording to embodiments of the disclosure; and

FIGS. 40A-40B illustrate exemplary plant level alarm aggregation by anHMI according to embodiments of the disclosure.

Identical reference numerals have been used, where possible, todesignate identical elements that are common to the figures. However,elements disclosed in one embodiment may be beneficially utilized onother embodiments without specific recitation.

DETAILED DESCRIPTION

This description and the accompanying drawings illustrate exemplaryembodiments of the present disclosure and should not be taken aslimiting, with the claims defining the scope of the present disclosure,including equivalents. Various mechanical, compositional, structural,electrical, and operational changes may be made without departing fromthe scope of this description and the claims, including equivalents. Insome instances, well-known structures and techniques have not been shownor described in detail so as not to obscure the disclosure. Furthermore,elements and their associated aspects that are described in detail withreference to one embodiment may, whenever practical, be included inother embodiments in which they are not specifically shown or described.For example, if an element is described in detail with reference to oneembodiment and is not described with reference to a second embodiment,the element may nevertheless be claimed as included in the secondembodiment.

Referring now to FIG. 1, an example of an industrial plant assethierarchy 100 is shown of the kind that may be created in accordancewith embodiments of the present disclosure. The term “plant asset” asused herein generally refers to any plant equipment, instruments,groupings thereof, devices associated therewith, and the like, that arecommonly employed in an industrial plant. At the bottom of the assethierarchy 100 are instruments (0) 102, such as sensors, monitors,actuators, and the like. Immediately above the instruments 102 areequipment (D) 104, such as vessels, heaters, exchangers, pumps, motors,mixers, and the like. One or more instruments 102 may connect to one ormore equipment 104 to form either a closed or an open control loop 106that performs some subprocess in the plant process, such as mixing,heating, boiling, and the like. Several types of equipment 104 andinstruments 102 may combine to form a unit (U) 108 that serves somefunction within the plant process, such as a gas synthesizer unit.Multiple such units 108 make up an area (A) 110, while multiple areas110 make up a plant site (S) 112, and multiple sites 112 form anenterprise (N) 114.

Additionally, in general, while each node in the hierarchy 100 can havechildren and parent, each child can usually have only one parent.However, assets like pipes and wires can be treated as equipment so thata pipe carrying material from/to two assets can be logically split intotwo pipes. A device or instrument can only be connected to oneequipment. Such a hierarchy 100 arranges and organizes plant assets in away that is logical and much easier for plant control systems and plantoperators to understand and use.

FIG. 2 shows an exemplary engineering drawing or diagram 200 that may beused to build the asset hierarchy 100 (or portions thereof) according toembodiments of the present disclosure. The engineering diagram 200 maybe any diagram, such as a process flow diagram (PFD) or a piping andinstrumentation diagram (P&ID), that graphically depicts how variousinstruments 102 (e.g., sensor) and equipment 104 (e.g., vessels) in theplant are connected to one another. The exemplary diagram 200 in thisexample is a P&ID, each P&ID displaying a plurality of symbols thatrepresent devices or instruments, equipment, and other componentstypically needed for implementing one unit 108. Connectors 202 locatedthroughout the diagram 200 specify how individual PI&Ds (i.e., units)are connected together to form an area, site, and so forth. Each diagram200 typically also has an information block 204, usually in the lowerright-hand corner, that displays a drawing identifier 206, a unitidentifier, and other information (e.g., revision number) that helpuniquely identify the PI&D. Other well-known parts of the PI&D notexpressly mentioned here include tag numbers, lines (e.g., process,electrical, data link, etc.), loops, and the like.

FIG. 3 illustrates an exemplary industrial plant control system 300 thatcan be used to process the engineering diagrams 200 and other plantengineering data sources according to embodiments of the presentdisclosure. As can be seen, the exemplary system 300 includes one ormore processors 302, an internal input and/or output (“I/O”) interface304, and a memory 306, all communicatively coupled and/or electricallyconnected to each other. These components allow the system 300 to, amongother things, extract various instruments 102, equipment 104, controlloops 106, units 108, areas 110, and sites 112 from the engineeringdiagrams 200, lines (e.g., process, electrical, data link, etc.),establish relationships between the asset to build a plant assethierarchy 100, generate an ontological knowledge base from the plantasset hierarchy 100, and provide an HMI for controlling the plantontological knowledge base.

The system 300 may be communicatively coupled and/or electricallyconnected to an external I/O component 308. The external I/O component308 allows the system 300 to interact and communicate with users andexternal systems. The communication may be accomplished over one or morecommunication networks (not expressly shown) connected to the I/Ocomponent 308. The communication networks may include a wide areanetwork (WAN) and/or a local area network (LAN) that is connectable toother telecommunications networks, including other WANs, LANs, and/orportions of the Internet or an intranet. The communication networks maybe any telecommunications network that facilitates the exchange of data,such as those that operate according to the IEEE 802.3 (e.g., Ethernet)and/or the IEEE 802.11 (e.g., Wi-Fi) protocols, for example. In anotherembodiment, the communication networks are any media that allow data tobe physically transferred through serial or parallel communicationchannels (e.g., copper wire, optical fiber, computer bus, wirelesscommunication channel, etc.).

Operation of the various components of the system 300 is generally wellknown in the art and thus is only briefly mentioned here. Processors 302may be adapted to execute processor-executable instructions stored inthe memory 306. The memory 306 may be adapted to provideprocessor-executable instructions to the processor 302 upon request.Included among the instructions stored on the memory 306 is anindustrial plant control application 310. The industrial plant controlapplication 310 may include a number of functional modules that worktogether to extract assets, establish asset relationships, build anasset hierarchy, generate a knowledge base, and provide an HMI, amongother things. These functional modules may include, for example, animage converter 312, filtering algorithms 314, symbol extractor module316, tag pre-processing algorithms 318, tag extractor module 320, rulesengine 322, line extractor module 324, intelligent processing module326, and an HMI application 328.

Image converter 312 is configured to convert diagrams 200 to an imageformat. In some embodiments, image converter 312 obtains diagrams 200 ina Portable Document File (PDF) or other electronic data format andconverts the diagrams to another image format, such as Portable NetworkGraphics (PNG), Joint Photographic Experts Group (JPEG), GraphicsInterchange Format (GIF), and the like. In some embodiments, imageconverter 312 creates two image files, one for display and one forcomputation (e.g., by filtering algorithms 314, symbol extraction 316,tag pre-processing 318, tag extractor 320, rules engine 322, lineextractor 324, and intelligent processing 326).

Filtering algorithms 314 are configured to process the compute image toobtain an approximate size of the symbols therein. Exemplary symbolsinclude, but are not limited to, those that conform to the InternationalSociety of Automation (ISA) standards for instruments, control/displayelements, programmable logic controllers (PLCs), valves, pumps, and thelike. In some embodiments, the symbols include identification letters(e.g., “FIC”) and a tag number (e.g., “123”). Obtaining the approximatesize of the symbols helps normalize the symbols for machine learningpurposes (via intelligent processing 326), as discussed later herein(e.g., to avoid creating training data for different sizes of symbols).

Symbol extractor 316 is configured to detect the symbols extracted fromthe images. In some embodiments, symbol extractor 316 applies imageprocessing algorithms to identify probable regions of symbols in theimages, then detects the symbol types and locations in the images via agross symbol identification technique. The symbol extractor 316maintains a running count of newly detected symbols in order to keeptrack of the number of detected symbols and determine whether any newsymbols were detected during a given execution cycle.

Tag pre-processing 318 is configured to remove symbol lines fromdetected symbols in the compute image, leaving only the tag components.In some embodiments, this involves centering the symbol, then removingthe symbol lines from the symbols, leaving only the tag components.Connected pixels are clustered and anything less than a standard textsize and greater than a standard text size is removed. Each cluster ofpixels is assigned a bounding box that defines a boundary around thecluster for processing purposes. Tag pre-processing 210 then findsbounding boxes at the same level vertically and in order from left toright. This allows tag pre-processing 210 to remove non-tag pixels andnoise.

Tag extractor 320 is configured to extract the tag component of a symbolin the compute image, such as a tag name and tag number. In some cases,neighboring characters in the tag name and/or tag number are joined witheach other and should be separated. In these cases, tag extractor 320checks for vertical gaps in the characters of the tag and segments thecharacters and thereafter performs character recognition using machinelearning techniques (via intelligent processing 326). When no verticalgaps are present, tag extractor 320 determines whether a width-to-heightratio of the given character set is greater than a predeterminedthreshold value (e.g., 0.6, etc.). If the width-to-height ratio isgreater than the predetermined threshold value, tag extractor 320applies segmentation using pixel density in the vertical direction.Areas that show peaks of white pixels are potential areas of split injoined characters. Thereafter, tag extractor 320 performs characterrecognition using machine learning techniques.

Rules engine 322 is configured to verify extracted tags from the computeimage based on one or more rules. In some embodiments, the rules arebased on ISA symbol standards and are divided into two categories: majorcompliance checks (e.g., red category) and minor compliance checks(e.g., orange category). Exemplary major compliance checks include, butare not limited to, verifying that the symbol is one of the valid types(e.g., field device, control room display, etc.) and verifying that thetag name has one or more identification letters. Exemplary minorcompliance checks include, but are not limited to, verifying thatidentification letters in a tag name do not contain any numerical digitsand the tag number in a tag name does not contain any alphabetcharacters except at the end.

The line extractor 324 is configured to extract lines between symbols inthe compute image. In some embodiments, the extracted lines comprisepiping and connections symbols, such as, piping, process connections,electrical signals, pneumatic signals, data links, capillary tubing forfilled systems, hydraulic signal lines, and guided electromagnetic orsonic signals. As will be understood by one of ordinary skill in theart, lines are extracted from the image using geometrical line fittingalgorithms. Once lines are extracted, a portion of the line is subjectedto one or more machine learning models (via intelligent processing 326)to obtain the type of the line as mentioned above. Additional detailsregarding operation of modules 312-324 may be found in U.S.Non-Provisional application Ser. No. 16/021,867 mentioned above andincorporated herein by reference. Intelligent processing 326 and HMIapplication 328 are discussed further below.

FIG. 4 shows an exemplary implementation of intelligent processing 326according to embodiments of the present disclosure. As discussed, thismodule 326 implements machine learning models and algorithms to identifysymbols in the compute image. In some embodiments, intelligentprocessing 326 applies a deep neural network to specific areas of theimage obtained through advancing a sliding window over the entire image.In the FIG. 4 example, intelligent processing 326 includes a data inputcomponent 402, one or more machine learning models 404, an automatedfeedback/correction component 406, a user application 408, a manualfeedback/correction component 410, and an analyzer 412.

In general operation, the data input component 402 receives datarepresenting diagrams (e.g., P&IDs, PFDs, etc.) and, after appropriatepre-processing, feeds the data to the one or more machine learningmodels 404. The machine learning models 404 use machine learning andimage processing techniques to extract relevant information, such asnames, numbers, symbols, lines, loops, and the like from the input data.The machine learning models 404 may also use image processing and/orgeometrical algorithms to reduce noise and enhance accuracy. Theautomated feedback/correction component 406 applies rules and algorithmsconfigured to detect errors in the output received from machine learningmodels 404. These errors are used to auto-correct the model output andfed back to the machine learning models 404 via the analyzer 412 tothereby update the learning of machine learning models 404. Theprocessed output from automated feedback/correction component 406 isthen displayed to a user for validation via the user application 408(e.g., HMI application 328). The corrections made by the user arecaptured by the manual feedback/correction component 410 and fed backinto the machine learning models 404 via the analyzer 412 to update thelearning of machine learning models 404. In this manner, intelligentprocessing 326 can continuously evolve and improve evaluation of inputdata and extraction of relevant information therefrom.

As mentioned, intelligent processing 326 applies a deep neural networkin some embodiments to specific areas of the image obtained. The deepneural network processing results in a multi-class classification ofsymbol candidates. In some embodiments, the symbols are classified perISA symbology. Exemplary symbols include, but are not limited to,instruments, control/display elements, programmable logic controllers(PLCs), valves, pumps, and the like. In some embodiments, intelligentprocessing 326 utilizes at least three types of convolutional neuralnetworks to recognize the various symbol candidates. The three typesinclude a decider network to decide if the input is single or multiplecharacters, a single character network to recognize single alphabet andnumeral characters, and a multi-character network to recognize multiplecharacters or words. In some embodiments, intelligent processing 326also utilizes context-based prediction to differentiate between similarcharacters, such as the capital letter “I” from the number “1” and thenumber “0” from the letter “0” and the like.

FIG. 5 an exemplary implementation of the HMI application 328 accordingto embodiments of the present disclosure. The HMI application 328generally begins at block 504 with inputting data from various plantengineering data sources, including plant engineering diagrams (e.g.,PI&D, PFD, etc.), into the industrial plant control system 300. At block506, the data is processed to extract relevant information and assets,such as names, numbers, symbols, lines, loops, and the like from thedata. The asset extraction is done using machine learning viaintelligent process 316 and the ML models therein. Processing the datausing the ML-based asset extraction described herein constitutes apractical application (e.g., the design of an HMI for an industrialplant using a specialized automated design system). Processing systemsand methods described herein may improve the process of HMI design bypermitting an image recognition model that identifies plant componentsin a diagram to run more efficiently and accurately. This can beachieved by inputting images that were predetermined by thepre-processing module as likely to lead to a positive identification bythe image recognition ML model. This may also permit better training ofthe model, thus improving the accuracy of the model.

At block 508, extracted assets and other relevant information are usedas input to an asset relationship establishing process to build an assethierarchy, as described in more detail later herein. The assetrelationship establishing is also done with the aid of intelligentprocess 316 and the ML models therein. Building an asset hierarchy usingthe ML-based asset relationship building process described herein is notwell-understood, routine, or conventional in the field of HMI design.Building an asset hierarchy using the methods and systems describedherein constitutes a practical application (e.g., the design of an HMIfor an industrial plant using a specialized automated design system).

At block 510, the asset hierarchy may be used to create a knowledgegraph based on a semantic model, as described in more detail laterherein. Creating a knowledge graph using the methods and systemsdescribed herein is not well-understood, routine, or conventional in thefield of HMI design. Creating a knowledge graph using the methods andsystems described herein constitutes a practical application (e.g., thedesign of an HMI for an industrial plant using a specialized automateddesign system).

At block 512, the knowledge graph may be used to generate an HMI assetmodel (e.g., automatically using a computer), as described in moredetail later herein. Generating a knowledge graph using the methods andsystems described herein is not well-understood, routine, orconventional in the field of HMI design. Generating a knowledge graphusing the methods and systems described herein constitutes a practicalapplication (e.g., the design of an HMI for an industrial plant using aspecialized automated design system).

At block 514, the HMI asset model may be used to provide or build an HMI(e.g., automatically using a computer), as described in more detaillater herein. Generating the HMI using the methods and systems describedherein is not well-understood, routine, or conventional in the field ofHMI design. Providing an HMI using the methods and systems describedherein constitutes a practical application (e.g., the design of an HMIfor an industrial plant using a specialized automated design system).

FIG. 6 shows the exemplary ML-based asset extraction process 506 and theML-based asset relationship building process 508 from FIG. 5 in moredetail.

ML-based asset extraction 506 generally begins at block 602 whereengineering diagrams (e.g., PI&D, PFD, etc.) are converted from PDF toimage format. At block 604, a user inputs information identifying aplant site and an area where each diagram is being used. At block 606,all text in the diagrams are found, and at block 608, unit identifiersfor the diagrams are found.

In some embodiments, finding a unit identifier involves searching in theinformation block of a diagram or other predefined portion of thediagram for certain keywords, such as “Drawing” or “Unit” or “Section”or variations thereof. The predefined portion may be identifiedautomatically based on analysis of previous diagrams (e.g., therightmost area having a width that is 20% of the width of the P&IDs andPFDs, and the lowest area having a height that is 20% of the height ofthe P&IDs and PFDs), or the portion may be specified by the user. Uponfinding the keyword (e.g., “Unit”), the system checks for textassociated with the keyword, which may be in the same cell as the word“Unit” if there is a table, or to the right of the word, or below theword. Once the unit label has been determined, the determined label maybe displayed to a user for validation. In some embodiments, this processmay be performed for a relatively small number of diagrams (e.g., about20). In this case, the positions of the determined labels may be used todetermine subsequent labels in other diagrams without requiring orsuggesting validation of the determination.

At block 610, all devices and instruments in a diagram (e.g., tags,valves, sensors, actuators, etc.) are found for each diagram, and atblock 612, all found devices and instruments as well as text are removedor otherwise suppressed from the diagrams. This process may comprisegenerating a bounding box around diagram features that are to be removedand setting all internal pixels to the background color of the diagrams(e.g., white or black). The removal of text and devices makes it easierto find all the lines in the diagrams at block 614.

In some embodiments, the lines in the diagrams may be found by scalingdown the diagrams and converting them to gray scale. Long horizontal andvertical lines may then be removed by detecting and deleting blackpixels (in the case of a white background) that extend along the X and Yaxes for longer than a predetermined length (e.g., 0.7 times the widthand height, respectively, of the diagrams). Once this is done, theremaining horizontal and vertical lines may be found by searching forclusters of black pixels (in the case of a white background). A clusteris a group of pixels separated by less than a predetermined number ofpixels (e.g., less than four pixels). Once found, the clusters may beconnected to create horizontal and vertical lines by combining all theco-linear points of the cluster. Lines that are smaller than apredetermined length (e.g., 16 pixels) may be removed. In embodimentswhere the diagrams are scaled before lines are removed, the removedlines may be scaled back to their original size. This may permit theoriginal line coordinates to be stored and referenced when determiningthe connections and relationships between plant components. Various linetypes may be identified, including piping and connections symbols, suchas piping, process connections, electrical signals, pneumatic signals,data links, capillary tubing for filled systems, hydraulic signal lines,and guided electromagnetic or sonic signals. In some embodiments, theabove line extraction can be done using geometrical line fittingalgorithms.

Once the lines are extracted, a portion of the line is subjected to oneor more machine learning models to obtain the type of the line asmentioned above. In some embodiments, a line extractor comprisesprocessor-executable instructions embodied on a storage memory device toprovide the line extractor via a software environment. For example, theline extractor may be provided as processor-executable instructions thatcomprise a procedure, a function, a routine, a method, and/or asubprogram.

Continuing at block 616, all arrows in a diagram may be found for eachdiagram. Such arrows indicate flow direction in the diagrams. Anexemplary technique for identifying arrows in a diagram is describedwith respect to FIGS. 7A and 7B.

Referring to FIGS. 7A and 7B, an exemplary arrow 702 is shown. Toanalyze arrow 702, in some embodiments, a detected line 704 as shown inFIG. 7A is assigned a bounding box 706 that has, for example, a widthperpendicular to the direction of the line that is a predeterminednumber of pixels wide (e.g., 40 pixels). To determine whether the line704 is an arrow, two squares 708, 710 on each end of the line may bedrawn with side lengths equal to the height of the generated boundingbox 706. In some embodiments, the sides of squares 708, 710 may havedifferent lengths. The contents of squares 708, 710 may be analyzedusing a machine learning model or algorithm trained to determine thepresence of an arrowhead 712 and the direction of the arrowhead. The MLalgorithm may be, for example, a Convolutional Neural Network. Thetraining data for the algorithm may be created by cropping head and tailportions of identified line segments by a specific size (e.g., 40×40pixels). These cropped images may be stored and later classified as aleft arrow, a right arrow, an up arrow, a down arrow, or no arrow. Thisclassification may be done by, for example, a human. Identified linesegments may be selected from diagrams received from different sources(e.g., different clients). In some embodiments, about 200 trainingimages may be used for training. If both squares 708, 710 have anarrowhead, arrow 702 may be considered bidirectional. If only one of twosquares 708, 710 has an arrowhead, arrow 702 may be consideredunidirectional.

Referring back to FIG. 6, after arrows are identified, all equipment ina diagram (e.g., tanks, pumps, boilers, heat exchangers, etc.) are foundfor each diagram at block 618. The equipment is then assigned a namethat helps uniquely identify the equipment. One of several namingconventions may be used depending on the particular industry. Forexample, the following naming convention may be used: <Type ofEquipment>-<Name/ID of Equipment>, where hyphens or spaces act asdelimiters. In the example of FIG. 2 above, the name assigned to thevessel may be “C-1119” where C is the type of equipment, and 1119 is theidentifier for the equipment within the diagram. This name may then bepersisted as the Regex (Regular Expression) for the tank for searchingpurposes. Similar naming conventions may be applied to other diagramcomponents.

In some embodiments, finding the equipment combines a geometrical orgeometry-based identification approach with machine learning or ML-basedclassifications.

Geometry-based identification of equipment is performed by firstremoving all instruments (e.g., by backgrounding the instruments) fromthe diagram to simplify the process, then finding all parallel linesthat satisfy one or more criteria. Paths joining the parallel lines arethen found and used to connect pairs of such parallel lines. A tank, forexample, may be found by searching for parallel lines and connectingthose pairs of lines that fulfill one or more criteria. A bounding boxmay then be defined around each pair that are thus connected. Onecriterion may require that two lines be within a predetermined distanceof each other. In some embodiments, the predetermined distance may bebetween about 10% and 120% of the greater length of the two lines. Theminimum length of such lines may be set at about 4% of the width of thediagram. The maximum length of such lines may be set at about 50% of thewidth of the diagram. The difference between the lengths of such twolines may be less than about 15% of the greater length of the two lines.The start and end points of the two lines (i.e., four points) may beused to make a box. The width and/or the height of the box may beexpanded by a predetermined percent. For example, if the width-to-heightratio is less than about 0.3, the width may be expanded by about 80%. Ifthe width-to-height ratio is between about 0.3 to about 0.4, the widthmay be expanded by about 60%. If the width-to-height ratio is more thanabout 0.4, the width may be expanded by about 40%. If the box is to beexpanded beyond the figure boundary, the expansion may be stopped at thefigure boundary. The resulting expanded box may be the bounding box. Theimage of the tank may be cropped at the bounding box. In someembodiments, the cropped image boundaries may be erased by making, forexample, about 4 pixels of one or more sides white (in the case of awhite background).

ML-based classification may then be used to identify the equipment type.For example, a machine learning model or algorithm may be used toanalyze the resulting image for the presence of a tank and identify thetank type. The ML model or algorithm may be trained to detect andclassify tanks.

In some embodiments, a resulting image may be analyzed to determinewhether it is to be inputted into the model. For example, the image maybe inputted into the model if it has a cluster comprising connectedpixels (e.g., pixels with a maximum gap of about 2 pixels) having anexpanded bounding box with a maximum dimension greater than the maximumdimension of the cluster's non-expanded bounding box and comprising thestart and end points of the two parallel lines identified earlier in thetank-identification process. In some embodiments, the expanded boundingbox may be generated by extending one or more lengths of thenon-expanded bounding box based on the width-to-height ratio identifiedearlier in the process. The images inputted into the model may be, forexample, 170×170 pixel black-and-white images. The model may be trainedwith a training data set of, for example, 1950 images. Instead or inaddition, augmented training data may be used. For example, 46312 imagesmay be used for training and 2518 for validation. Table 1 below showsexemplary training-data augmentation details as would be understood bythose having ordinary skill in the art.

TABLE 1 Exemplary Training-Data Augmentation Augmentation MethodsDescription Offset-Value Zero-offset Project input original 0 image to64 × 64 pixel- size image Horizontal and Offset horizontal andHorizontal: vertical offsets vertically zero-offset 2, 4, 6, −2, −4, −6image with specified Vertical: values 2, 4, 6, −2, −4, −6 Diagonaloffset Move zero offset Diagonal values: diagonally (−2, −2), (−2, 2),(2, −2), (2, 2), (4, 4), (−4, −4), (−4, 4), (4, −4), (6, 6), (−6, −6),(−6, 6), (6, −6)

Those skilled in the art will also understand that the above table, or asimilar table, may be used to provide training-data augmentation for anyof the other ML-based processes described herein.

Referring now to FIG. 8, other equipment, such as pumps, may also beidentified. Pumps may be identified by creating a bounding boxcontaining the endpoint of one or more horizontal lines in a diagram.FIG. 8 shows an exemplary pump 800 around which a bounding box may becreated. To create the bounding box, there may be traversal of pixelsfrom an end point (e.g., P1) of horizontal line 802 in each directionperpendicular to line 802 until a black pixel (e.g., P2) is found. Thedistance (e.g., D) between P1 and P2 may be computed. If the distance isbetween predetermined values (e.g., 100 pixels and 200 pixels) then abounding box may be created such that P1 and P2 are at the centers ofthe opposite sides of the bounding box. The width of the bounding boxmay be equal to the distance D. The bounding box may be expanded by 20%of the length towards point P2. The diagram may be cropped at thebounding box. Clusters of connected pixels may be generated from thecropped image by creating the groups of pixels with each group havingall the pixels connected together horizontally, vertically ordiagonally. For clusters with a length or width above a predeterminedthreshold (e.g., 150 pixels), the areas for the clusters may bedetermined. The cluster with the greatest area may have its bounding boxdetermined by, for example, finding the minimum and maximum values of xand y coordinates in the cluster. The minimum values of x and ycoordinates may determine the top left corner of the bounding box. Themaximum values of x and y coordinates may determine the bottom rightcorner of the bounding box. The image may be cropped at this cluster'sbounding box. The resulting image may be analyzed for the presence of apump and its type using, for example, a machine-learning algorithmtrained to detect and classify pumps. For example, a CNN architecturesimilar may be used.

Returning again to FIG. 6, at block 620, connectors in diagrams andtheir associated information may also be identified. Recall from abovethat connectors indicate connections between components located ondifferent diagrams.

FIG. 9 illustrates an exemplary connector 902. As can be seen, rear end903 of connector 902—the end without an arrowhead—has a line 904connected to it, indicating that flow occurs from a component identifiedby a source identifier 906 over line 904 and to a component identifiedwith a destination identifier 908 over connector 902. A connector with aline connected to its front end 909—the end with the arrowhead—mayindicate that flow occurs from a component identified by a sourceidentifier over the connector and to a component identified with adestination identifier over the line. A connector identifier 910 withinconnector 902 may indicate a diagram identifier (unit label) of thediagram containing the component identified with destination identifier908. Similarly, the connector identifier 910 within the connector 902may indicate the diagram identifier of the diagram containing thecomponent identified with the associated source identifier. Connectorsmay be identified by filtering identified lines having a length betweenabout 20 pixels to about 40 pixels. Pairs of lines with start pointsless than, for example, four pixels apart, end point less than, forexample, four pixels apart, may be analyzed to determine whether theyare part of a connector. If the length of one line in a pair is greaterthan the length of the other line in the pair, and the distance betweenthe pair is more than about 0.1 and less than about 0.5 times the lengthof the longer line, the pair may be analyzed to determine whether theyare part of a connector. A tight-fitting bounding box containing bothlines may be created and each side expanded by half the distance betweenthe two lines. Three resulting exemplary connector images areillustrated FIGS. 9A, 9B, and 9C respectively at 912, 914, and 916. Theresulting images may be inputted into a machine-learning model trainedto identify the presence and direction of a connector.

Note that the source identifier 906 may be highlighted or otherwiseemphasized for easier viewing by the user. This technique may be appliedthroughout THE HMI to call the attention of the user to a particularfeature.

To identify information associated with connector 902, a bounding box(not expressly shown) may be generated around text 910 inside connector902 to determine if it is a connection identifier. If the bounding boxis contained by the bounding box of connector 902, the text may beidentified as connection identifier 910. Text situated above thebounding box of connector 902 within, for example, four times thedistance between parallel lines of connector 902 (e.g., the two parallellines connected to the arrowhead), may be identified as destinationidentifier contained in 908. In some embodiments, instead or inaddition, text before and after the word “To” may be identified as amaterial name and destination identifier 908, respectively. In thiscontext, the material name may identify the material flowing through apipe or other material carrier. In some embodiments, instead or inaddition, text before and after the word “From” may be identified as amaterial name and a source identifier, respectively. To identify thepipe identifier contained in 906, strides from the bounding box ofconnector 902 may be taken in the direction of line 904. In someembodiments, the strides may be two times the length of the bounding boxof connector 902. When the shifted connector bounding box intersectswith a bounding box of a word, the word may be identified as the pipeidentifier contained in 906. If bounding boxes of multiple words areintersected, the word with the largest number of characters may beidentified as the pipe identifier contained in 906. If connector 902 hasa line connected to its front end, one or more of the preceding methodsmay be used to identify a source identifier and/or a destinationidentifier.

Returning once again to FIG. 6, after the ML-based asset extractionprocess 506 has concluded, the ML-based asset relationship buildingprocess 508 can begin for the extracted assets. This process of buildingrelationships between extracted assets may comprise creating lineconnectivity graphs that capture the connections illustrated in thediagrams (e.g., PI&D, VFD, etc.), then merging the graphs to create acomposite flow graph. The graphs that are merged may comprise graphsrepresenting diagram lines connected to equipment, graphs representingdiagram lines connected to connectors, and/or graphs representingdiagram lines connected to instrument components (e.g., tags).Additionally, line connectivity, such as a line-line (line-to-line)graphs may also be generated to facilitate determining how variouscomponents are connected.

At block 622, a line-line (line-to-line) graph may be generated bydetecting lines connected to a component's bounding box and checkingsuch lines for co-linearity or perpendicularity. If co-linearity orperpendicularity is detected and the distance between the start or endpoint of one line and the start or end point of another line is lessthan a predetermined value (e.g. 32 pixels), the lines may be extendedso that their ends meet. Lines that form part of a component (e.g.,equipment) may be removed by removing those lines that have bothendpoints lying within the component s bounding box. To assign indicesto line segments and to keep track of line connections, lines may besplit. Splitting a line into two segments is particularly useful if adevice (e.g., tags, valves, etc.) is lying on top of the line. To dothis, the distance to the start and end points of vertical lines may bemeasured from horizontal lines. In some embodiments, the start and endpoints of the horizontal lines may be excluded from this measurement. Ifthe measured distance is less than a predetermined distance (e.g., 32pixels), a split may be made in the horizontal line at the closest pointbetween the horizontal line and the start or end point of the verticalline. This procedure may be repeated for vertical lines; if the distancefrom a vertical line to a start or end point of a horizontal line isless than a predetermined number (e.g., 32 pixels), a split may be madein the vertical line at the closest point between the vertical line andthe start or end point of the horizontal line. In some embodiments, thedistance from the vertical line to a start or end point of thehorizontal line may exclude the start and end points of the verticalline. A line may be assigned an index number. When a line is split at apoint, the two newly created segments may be assigned an index number.An adjacency matrix may be generated that represents line connections.

FIG. 10A illustrates an exemplary adjacency matrix 1002. Adjacencymatrix 1002 may have a size of n×n, where n is the total number oflines. The value at position [first line index, second line index] inadjacency matrix 1002 may be “0” to indicate that the lines with thefirst and second line index, respectively, are not connected to eachother. The value at the position may be a “1” to indicate that theselines are connected. It is to be understood that other values may beused. Thus, adjacency matrix 1002 can indicate whether two lines areconnected to each other. For example, going from left to right, node L2has a “1” at node L1 and node L3 to indicate that L2 is connected to L1and L3. To build the line-line graph, a node-edge (node-to-edge) graphmay be generated where line indices are assigned a node and connectedlines indices are assigned edges between nodes.

FIG. 10B illustrates an exemplary line-line graph 1004 corresponding tothe adjacency matrix 1002. In the figure, nodes E1 and E2 represent twopieces of equipment and nodes L1, L2, L3, L4, and L5 represent linesegments. As can be seen, line segments L1, L2, L3, L4, and L5 connectE1 and E2 together in the manner reflected by the adjacency matrix 1002.

Returning to FIG. 6, at block 624, graphs representing equipment-line(equipment-to-line) connections may be generated from the diagrams aspart of the ML-based asset relationship building process 508. An exampleof building an equipment-line graph according to some embodiments isshown in FIGS. 11 and 12A-12B.

FIG. 11 shows an exemplary piece of equipment 1102. As can be seen, abounding box 1104 has been placed around the equipment 1102 forcomputing purposes. To build an equipment-line graph, the bounding boxis expanded on each side by increasing the length of each side, forexample, by 2.5%. Lines 1106 with end points lying in the expandedbounding box are then found and an associated node (e.g., L1, L2, L3,etc.) is generated for each line. In some embodiments, if two such linesare connected with each other, the line with an end point closer to thecenter of the bounding box may have an associated node generated. Theequipment 1102 may also have an associated node (e.g., E1, E2, E3, etc.)generated. Thereafter, equipment-line pairs may have associated edges(e.g., (E1, L1), (E2, L2), (E3, L3). etc.) generated.

FIG. 12A illustrates exemplary tables showing equipment-lineconnections. The figure shows an equipment-line table 1202 listing thevarious nodes (e.g., L1, L2, L3, etc.) and (e.g., E1, E2, E3, etc.)associated with the equipment 1102 and the line segments 1106, and anedges table 1204 listing the edges (e.g., (E1, L1), (E2, L2), (E3, L3),etc.) associated with those nodes.

FIG. 12B illustrates an exemplary equipment-line graph 1206corresponding to the equipment-line table 1202 and edges table 1204. Ascan be seen, there are five edges, Edges 1, 2, 3, 4, and 5, reflectingthe five points where lines L1, L5, L6, L7, and L9 are connected toequipment E1, E2, E3, and E4.

To find a path between two pieces of equipment, lines connected to thepieces of equipment may be identified by merging the graph 1206representing diagram lines connected to equipment with adjacency matrix1002. If multiple paths are found between the two equipment, then theshortest-path algorithm (e.g., Dijkstra's shortest-path algorithm) maybe used on the graph to identify the path between the two pieces ofequipment. The arrow direction associated with the path's lines mayindicate the direction of material flow between equipment and designatewhich equipment is the destination and which is the source.

Returning once more to FIG. 6, at block 626, graphs representingconnector-line (connector-to-line) connections may be generated from thediagrams as part of the ML-based asset relationship building process508. Following is an example of building a connector-line graphaccording to embodiments of the present disclosure.

To build a connector-line graph, a bounding box is placed around aconnector as discussed with respect to FIG. 9. The bounding box may thenhave its long sides expanded, for example, by 0.1 times the lengththereof. The expansion may be in the direction of the arrow if theconnector is an incoming connector, and in the opposite direction if theconnector is an outgoing connector. A line with an end point lyingwithin the expanded bounding box may then be identified. A node may thenbe generated for the connector, such as C1, and a node may also begenerated for the line connected to the connector, such as Lc1. Adirected edge may be then generated for the connector-lines pair, suchas (C1, Lc1), with the edge direction determined by whether the line isconnected to the connector's head or tail. To determine whetherequipment is connected to a connector, a shortest-path algorithm (e.g.,Dijkstra's shortest-path algorithm) may be used on the graph todetermine a path between a line connected to the equipment and a lineconnected to a connector. If a connection is determined, the directionof the edge connected to the connector node can be used to determine thedirection of the flow between the equipment and the connector.

FIG. 13A illustrates exemplary tables showing connector-lineconnections. The figure shows a connector-line table 1302 listing thevarious nodes (e.g., C1, C2, Lc1, Lc2, etc.) associated with theconnectors and line segments, as well as an edges table 1304 listing theedges (e.g., (C1, Lc1), (C2, Lc2), etc.) associated with those nodes.

FIG. 13B illustrates an exemplary connector-line graph 1306 thatcorresponds to the connector-line table 1302 and edges table 1304. Ascan be seen, there are five connectors, C1, C2, C3, C4, and C-5connected to two pieces of equipment, E1 and E2, by line segments wherelines L1-L16. Table 1308 shows the paths that are generated using thegraph 1306.

Returning once more to FIG. 6, at block 628, graphs representing deviceor instrument-line (device/instrument-to-line) connections may begenerated from the diagrams as part of the ML-based asset relationshipbuilding process 508. Following is an example of building adevice/instruments-line graph according to embodiments of the presentdisclosure.

A graph representing lines connected to devices or instrument (e.g.,tags, sensors, etc.) may be generated by providing bounding box aroundthe device. Then, lines with end points lying within the bounding boxmay be identified. If less than two lines are found per bounding box,one or more sides of the bounding boxes can have their lengthsincreased, for example, by 50%. A node may then be generated for thedevice, such as D1, and node may also be generated for the lineconnected to the device, such as L1. Pairs of instruments and linesconnected thereto may have associated edges generated. To determinewhether an instrument and piece of equipment are connected, ashortest-path algorithm (e.g., Dijkstra's shortest-path algorithm) maybe used on the graph to determine a path between a line connected to theinstrument and a line connected to the equipment.

FIG. 14A illustrates exemplary tables showing device/instruments-lineconnections. The figure shows a device/instruments-line table 1402listing the various nodes (e.g., D1, D2, L1, L2, L3, L4, L5 etc.)associated with the devices and line segments, as well as an edges table1404 listing the edges (e.g., (D1, L1), (D1, L2), (D2, L3), D2, L4),(D2, L5), etc.) associated with those nodes.

FIG. 14B illustrates an exemplary device/instruments-line graphs 1406and 1408 that correspond to the connector-line table 1402 and edgestable 1404. As can be seen in graph 1406, device D1 has two linesegments, L1 and L2, connected thereto. In graph 1408, device D2 hasthree line segments, L3, L4, and L5, connected thereto.

Returning once again to FIG. 6, thus far, only directed graphs, orgraphs that have a particular flow direction, have been discussed. Inaddition to directed graphs, there may also be undirected graphs in thediagrams. Any such undirected graphs may be changed to directed graphsat block 630 as part of the ML-based asset relationship building process508. Following is an example of changing an undirected graph to adirected graph according to embodiments of the present disclosure.

To assign directions to a graph, such as a line-line graph,equipment-line graph, and device-line graph, all line nodes in anequipment-line graph are identified. For each of these lines, allpossible paths in the line-line graph are traversed up to the lines thatare connected to a connector or another piece of equipment. The pathsmay then be split into segments, and the lines in these segments may beassigned a direction. This may be done based on the direction of aconnected connector, the presence of an arrow, or the direction of linesin an adjacent path (e.g., lines connected to a connector). For example,if one path is A->B->C->D and another path is A->B->E, then split theseto paths into three paths: A->B, B->C->D and B->E. Bath directions maythen be assigned as follows. If one of the lines in a path is connectedto a connector, then assigned the direction of the connector (incomingor outgoing) to all lines in the path. If at least one of the lines inthe path has an arrow, then assigned the direction of the arrow to allthe lines in that path. If none of the lines in a path has an arrow andthe path is not connected to a connector, then check for an adjacentpath that is connected to a connector and assign the direction of theadjacent path to all lines in the path.

Returning once again to FIG. 6, recall from block 612 that all devicesand instruments (as well as text) were removed from the diagram,resulting in the line-line graphs showing only connections fromequipment to equipment and equipment to connectors. The line-line graphsmay now be updated to include those devices and instruments at block 632as part of the ML-based asset relationship building process 508. Thismay be done, for example, by restoring the devices to the point fromwhere they were removed.

FIG. 15A illustrates an exemplary line-line graph 1502 and edge datatable 1504 with the devices removed according to block 612 of FIG. 6. Ascan be seen, devices were removed from the points between the connector(C) and the equipment (E) where two dashed circles now sit and replacedwith two corresponding edges, Edge 5 (L3, L6) an Edge 6 (L6, L7).

FIG. 15B illustrates the same exemplary line-line graph 1502 and edgedata table 1504 after updating to restore the devices. As can be seen,two devices (D) are now positioned between the connector (C) and theequipment (E), replacing the two edges (L3, L6) and (L6, L7). Thisallows line connectivity graphs to reflect not onlyequipment-to-equipment and connector-to-equipment connections, but alsodevice-to-equipment, device-to-connector, and device-to-deviceconnections.

Returning once more to FIG. 6, at block 634, for a given diagram (e.g.,PI&D, PFD, etc.), the various line connectivity graphs generated forthat diagram may be merged as part of the ML-based asset relationshipbuilding process 508. For example, if the diagram is a PI&D, then allline connectivity graphs may be merged to create a PI&D flow graph.

FIG. 16A illustrates an exemplary PI&D flow graph 1602 resulting from amerger of the various line connectivity graphs generated for a givenPI&D by the ML-based asset relationship building process 508. This PI&Dflow graph 1602, unlike its constituent graphs, contains all theconnectors, devices or instruments, and equipment in the PI&D along withthe connections there between. The PI&D flow graph 1602 thus capturesthe relationship between all the assets that were extracted from thegiven PI&D.

FIG. 16B illustrates an exemplary table 1604 containing the data for themerged PI&D flow graph 1602. The table 1604 includes the nodes and edgesfor each of the line connectivity graphs (e.g., line-line,equipment-line, connector-line, device-line, etc.) that were generatedby the ML-based asset relationship building process 508 for the givenPI&D. The table 1604 also contains the nodes and edges of the mergedPI&D flow graph 1602. Such a table may then be used as part ofgenerating the HMI.

Returning once again to FIG. 6, at block 636, control loops for a givendiagram (e.g., PI&D, PFD, etc.) may be added to the merged PI&D flowgraph 1602 as part of the ML-based asset relationship building process508. As mentioned earlier, a control loop performs some sub-process inthe plant process and may be either a closed or an open control loop.Most control loops involve several devices working together, includingsensors, actuators, controllers, and the like. There are generally twotypes of control loops: simple control loops, and complex control loops.Simple control loops have an input, an output, and a controller, whilecomplex control loops are made of several simple control loops. Anexample of identifying a control loop according to embodiments of thepresent disclosure is depicted in FIG. 17.

FIG. 17 illustrates a portion of an exemplary diagram 1700 containing asimple control loop. Four symbols have been identified and labeled 1702,1704, 1706, and 1708 for illustrative purposes, each symbol representingeither a device (e.g., sensor, actuator, controller, etc.) or a piece ofequipment. Identifying a control loop in this example begins withrecognizing all tags inside the symbols and all words outside thesymbols. Next, for each word outside a symbol, the nearest tag insidethe symbol is found. For the symbol at 1702, for example, the nearesttag inside the symbol is “TIC 1803.” For a given symbol, all wordshaving a distance greater than, say, 1.2 times the width of the symbol(i.e., words associated with other symbols) can be ignored. Then, thewords are checked (e.g., using a rules engine) against a list of wordsfrom an industry standard like ISA to determine whether the wordsrepresent, for example, an alarm, a signal, a tag, or the like. If theword is a tag, then the tag number of the nearest tag is assigned tothat word. For the symbol at 1702, for example, the tag number “1803” isassigned to the words “TE” and “TT” to produce “TE 1803” and “TT 1803.”These tagged words are then added to the list of tags for the symbol.The tags for a given symbol include all the tags inside and outside thesymbol.

Once all tags inside and outside the symbols are recognized, all tagswith the same beginning identification letters and the same tag numberare grouped together. In the FIG. 17 example, the tags “TIC 1803,” “TV1803,” “TE 1803,” and “TT 1803” are grouped together. These tags areused to define a simple loop 1710 that can be included as an extractedasset for the diagram 1700.

Returning once more to FIG. 6, at block 638, plant assets are organizedin an asset registry for easy subsequent access and use. The assets areorganized using namespace in order to uniquely identify each asset. Asthose having ordinary skill in the art understand, namespace is a set ofnames used to identify objects so there is no ambiguity when objectshaving different origins but the same names are mixed together.

At block 640, the merged flow graph for each diagram may be merged aspart of the ML-based asset relationship building process 508 to create asingle plant flow graph for the entire plant. Recall that a diagram likea PI&D or PFD represents the instruments or devices and equipment makingup one unit, and that the asset extraction process discussed aboveresults in the equipment names for a given unit having the same unitprefix and therefore uniquely identifiable. The device names likewiseare created by prefixing the unit name. Connector names are alsouniquely identifiable because each connector name contains the name ofthe source equipment, destination equipment, and pipe name.

Line indices or index numbers (e.g., 1, 2, 3, etc.) are local to eachdiagram and thus need to be made uniquely identifiable across multiplediagrams. This can be done, for example, by prefixing the line indiceswith the unit name or the corresponding diagram followed by the drawingnumber. For example, if the unit name is “UNIT1” and the diagram drawingnumber is “1116,” then line index number “15” can be changed to“UNIT1_1116_15.”

To create a single plant flow graph for the entire plant, flow graphsfor each diagram in the plant may be generated as described above. Whereneeded, line indices may be made uniquely identifiable across thedifferent diagrams as described above. The flow graphs for the variousdiagrams may then be merged by inserting all nodes and edges from eachdiagram flow graph (or the table representations thereof) into the plantflow graph (or the table representation thereof). Diagrams that aredirectly connected to one another will have connector nodes with thesame names. When this occurs, the duplicate connector nodes are removedand a new edge is created in the plant flow graph by joining the linesthat were to be connected by the removed connectors. All other nodes andedges should remain the same.

The resulting plant flow graph may be used to generate a visualrepresentation of the asset hierarchy similar to the asset hierarchy 100shown in FIG. 1. The asset hierarchy and the relationships underlyingthe asset hierarchy may then be stored as structured data, for example,in one or more JSON (JavaScript Object Notation) files for subsequentaccess and use by the industrial plant control system 300.

In some embodiments, the industrial plant control system 300 may be usedto incorporate the structured data for the asset hierarchy and assetrelationships into an ontological knowledge base, as mentioned above.The system then dynamically builds a plant engineering domain ontologyfrom plant engineering data sources (which evolve over plant lifecyclephases). This generally involves: (1) extracting (by the system) thedomain entities (i.e., type and data) from the structured data sourcesusing metadata from unstructured data sources and machine learningtechniques; (2) building (by domain experts) a semantic model using theentity type and also add the association relationships between theentities; (3) dynamically extracting (by the system) the entity data andbuilding a knowledge graph based on the sematic model; and (4) providing(by the system) the ability for users to navigate the asset namespaceand relationships to extract relevant information. Such an arrangementhas a number of benefits.

As one benefit, automated control configurations can be performed usingprocess narrative. Currently, control configuration is a manual processthat is time consuming and requires strong control engineeringbackground. With the industrial plant control system 300 describedherein, a person who is not familiar with control engineering conceptsshould also be able to easily configure the system. This arises from thesystem's use of “process narratives” composed of ordinary language usedby plant engineers. The system automatically translates the processnarratives to control strategies and configurations. The processnarrative can be converted to multiple control narratives by leveragingnoun-verb pairs such as “Asset.Operation” in AutomationML (see IEC62714) and State Based Control (see ISA 106, ISA 88, and ISA 95). Thesystem translates process narrative to control logic using informationfrom the asset templates, their relationships, namespaces, controlstrategy templates and rules.

As another benefit, test cases can be automatically auto-generated basedon process narrative and documented validation. The industrial plantcontrol system 300 automatically generates the test cases using theprocess narratives, control configuration, and asset ontology. Thesystem then run the automated test scenarios to simulate the process inorder to validate the control application.

Referring now to FIG. 18, a flow chart is shown for an exemplary method1800 that may be used with the industrial plant control system 300 todynamically build a plant engineering domain ontology from plantengineering data sources. The method generally begins at block 1802where plant engineering data sources are input into the system. The datasources may include the PI&Ds and PFDs used to generate plant assethierarchy and asset relationships discussed above, as well as AutoCAD,SmartPlant, and Aveva Engineering diagrams. Other plant engineering datasources may include simulation models, instrument index, cableschedules, I/O allocation lists, configuration data, cross wiringtables, cabinet loading reports, system audit reports, historical data,alarms and events, and the like.

FIGS. 19-29 show exemplary plant engineering data sources that may beused to generate an asset hierarchy and/or a knowledge graph. Theseplant engineering data sources will appear familiar to those skilled inthe art. For example, FIG. 19 illustrates exemplary alarm data at 1900.FIG. 20 illustrates an exemplary cabinet loading report 2000. FIG. 21illustrates an exemplary control database at 2100. FIG. 22 illustratesan exemplary cross wiring report at 2200. FIG. 23 illustrates anexemplary field wiring index at 2300. FIG. 24 illustrates exemplaryhistorical data (tag data) at 2400. FIG. 25 illustrates an exemplaryinstrument index at 2500. FIG. 26 illustrates an exemplary assethierarchy at 2600. FIG. 27 illustrates an exemplary nest loading reportat 2700. FIG. 28 illustrates an exemplary plant asset index at 2800.FIG. 29 illustrates an exemplary process control narrative at 2900. Suchdata sources and other plant engineering data sources, including HMIspecifications (see FIG. 34), may then be used to generate an assethierarchy and/or a knowledge graph.

Returning to FIG. 18, at block 1804, the system extracts domain entitiesand instances for the domain entities from the plant engineering datasources. The data sources may include structured and unstructured datasources. For example, entities (also referred to as types or classes),properties (also referred to as attributes), and relationships may beextracted from the data sources. The entities may be used to specify theontology's domain and range, and values of properties may be used tospecify the ontology model's range. The semantic model may comprise aset of triples built from the ontology's domain and range. A triple mayspecify a subject entity, a predicate relationship or attribute, and anobject entity or value. The subject entity may be an entity from theontology's domain. The object entity or value may be an entity or valuefrom the ontology's range. Instances of entities and values may beextracted from the data sources. For example, the semantic model mayspecify a triple with subject “Unit,” predicate “has,” and object“Device.” A triple comprising instances of the corresponding subject andobject may include “Unit01” (an instance of entity “Unit”) “has” (thepredicate) “ControlValve01” (an instance of entity “Device”).

Multiple data extraction techniques can be used for each data source,both structured and unstructured, from the plant engineering datasources. The system can differentiate structured and unstructured databased on the file format or database type. Unstructured data files(e.g., email messages, word processing documents, videos, images,webpages, etc.) often include text and multimedia content without anyschema, while structured data files (e.g., spreadsheets, CSV files, XMLfiles, RDBMS, time series, graph database, etc.) contain a schema ormetadata. The system can extract domain entities from these structureddata sources using the schema or metadata. In a CSV file format, forexample, plain text data is separated by commas with each new line inthe CSV file representing a new database row and each database rowhaving one or more fields separated by a comma. The system can use thisschema to extract domain entities from the data source. An example canbe seen in FIG. 25, where domain entities were extracted from anexemplary instruments index data source in CSV format. Among the domainentities and associated relationships extracted from this data sourceare Plant, Area, Unit, Loop, Tag, Process Function, and the like, asindicated in the first row of the table. Instances of these domainentities are shown in the subsequent rows.

For unstructured data, such as P&IDs, PFDs, Process Control Narratives(PCNs) and other image or unstructured text formats, the system extractsdomain entities and associated relationships from the unstructured datasources into structured data using machine learning. For reference, aProcess Control Narrative (PCN) is a functional statement describing howdevice-mounted controls, panel-mounted controls, PLCs, HMIs, and otherprocessor-based process control system components should be configuredand programmed to control and monitor a particular process, a processarea, or facility. A PCN is the essential link between process designand control system design. For PCNs, the system uses Named EntityRecognition (NER) techniques to extract domain entities, such as pumps,valves, locations, alarm conditions, and the like, from the processnarratives. NER is a technique used in information processing withunstructured text. The technique labels sequences of words in the textthat are the names of things, such as things, person names,organizations, locations, time expressions, quantities, monetary values,percentages.

FIG. 30 is a table 3000 showing an example of domain entity types andinstance data extracted into a file having a tabular format. The domainentity types in the table include Controller, Compound, Block, Tag, andso forth, while the corresponding instance data include (for the firstrow) CP2801, CF101_011900, PT038, 011900PV038, and the like.

Returning to FIG. 18, once the domain entities and relationships havebeen extracted, then at block 1806, a user (e.g., a domain expert)builds a semantic model using the extracted entity types and associatedrelationships. FIG. 31 shows an example of a user interface that may beused by a domain expert to build a semantic model.

In FIG. 31, a user interface 2100 is shown that allows the user tospecify the semantic relationships between entities extracted from datasources. From this interface, the user can browse the entity typesmanually from various data sources and filter the entities required forthe semantic model. For example, the user can select two entities suchas Source and Target and create a named relationship therebetween. Theschema can be defined at conceptual level, which is a logical model. Theuser can then the domain specific relationships between the entitiesextracted from the data sources. Typical relationships are “is-a”representing a parent-child relationship, “has-a(part-of)” representinga containment relationship, and the like. In the example, the user hasspecified that a source “Plant” is related to a target “Area” by therelationship “has Area.” The user can also add additional contextualinformation as entity attributes. For example, the user can add Type,Severity, Priority as attributes for an Alarm entity.

In some embodiments, instead or in addition to being defined by a user,the semantic relationships may be determined automatically from, forexample, primary and foreign key relationships specified in RelationalDatabase Management System (RDBMS) tuples or parent-child nodes in XMLdocuments. Properties may be added to provide additional contextualinformation (e.g., for entity “Alarm,” the properties “type,”“severity,” or “priority” may be added). The resulting semantic modelmay be validated against a predetermined set of rules. The rules may bedefined by, for example, standard ISA-106 (“Procedural Automation forContinuous Process Operations”). Using ISA-106, a triple “Device hasUnit” may be flagged for correction during validation because thestandard specifies that Units have Devices rather than vice versa.Public ontologies and vocabularies in dictionary files may be used forvalidation instead or in addition to standards.

Returning to FIG. 18, once the semantic model has been built, then atblock 1808, the model is validated. Semantic validation refers to theprocess of verifying that the data elements are logically valid. Forexample, a data element such as Equipment-Has-Area would not be validunder the ISA 106 schema because equipment would not have an area(semantic models typically standardize the entity relation typesaccording to an industry standard). In some embodiments, the systemautomatically validates the domain specific relationships between theentities that were created in block 1806. To this end, the system canmaintain standard terms, their definitions, as well as references topublic ontologies and vocabularies in dictionary files, which can beused as a reference for validation. The system also provides the optionsto validate and modify the system-generated relationships by the domainexperts manually.

At block 1810, the validated semantic model is saved to create theontology. In some embodiments, the semantic model is saved according toone of the following well-known semantic model formats: OWL/XML,JOSO-LD, and RDF/JSON. As an alternative, the semantic model may bepersisted by converting the model into Labeled Property Graph Model inGaphSON format. If the semantic model is converted into W3C Ontology,then the semantic model knowledge graph can be persisted in RDF triplestores like AllegroGraph, Stardog or Amazon Neptune. If the semanticmodel is converted into LPG (Labeled Property Graph), then the knowledgegraph therefor can be persisted into Graph Database formats like GraphDBor Neo4j or Azure Cosmos DB.

At block 1812, the system may use the semantic model to build and deploya knowledge graph. As alluded to above, a knowledge graph is basically agraph that can be used to navigate and visualize the relationshipsbetween components (i.e., a physical model of the data). The semanticmodel defines the relationship between the source and target entities inthe knowledge graph at a conceptual level (e.g., “Area-has-Unit”). Thesystem extracts the instances related to the ontology classes and buildsthe knowledge graph using the associated relationships and thecontextual information. The knowledge graph thus represents a collectionof interlinked entities that enhances the ability of a user to searchfor desired information. The knowledge graph can be represented in thedatabase system using LPG, RDF, or similar graph models. An exemplaryvisualization of a knowledge graph is depicted in FIG. 32.

Referring to FIG. 32, a visualization of a knowledge graph 3200 can beseen in which entities and relationships are shown as a plurality oflabeled-property nodes, one of which is indicated at 3202. Both thenodes and their relationships are named and can store propertiesrepresented by key/value pairs. Nodes can be labeled so as to be easilygrouped with similar nodes. The edges representing the relationshipshave two main qualities: they always have a start node and an end node,and they are directed, making the graph a directed graph. Relationshipscan also have properties, which is useful in providing additionalmetadata and semantics to relationships of the nodes. A legend 3204provides an indication of what each node 3202 represents on acolor-coded basis, for example, blue for tag index list, green for nestloading report, orange for cabinet loading report, and so forth. Theknowledge graph 3200 can be selected by choosing a graph view option3206 and is one of several display screens that are available from anHMI according to embodiments of the present disclosure. Users can selectother display screens from the HMI by choosing the option therefor, suchas an upload data option 3206 to upload data and a query engine option3208 to search for data.

FIG. 33 illustrates an example of a query display screen 3300 availablefrom the HMI according to embodiments of the present disclosure. In thisscreen, the user can search for data by typing natural language queriesinto a search box 3302. For example, the user may ask that the HMIdisplay or otherwise indicate the alarms that typically are triggeredduring a normal plant startup process by entering the query “What arethe expected alarms during a startup sequence?” into the search box3302. In response, the HMI may display the requested alarms eithergraphically (nodes with dashed circles) or in text form by displayingthe alarms' identifiers. The user can also select and drag nodesrepresenting the type of data the user desires into the search box 3302.In the example shown, the user can drag and drop the six alarms thatwere returned into the search box 3302. By virtue of the knowledgegraph, the HMI can quickly highlight all common assets for the sixalarms, indicated by the nodes without dashed circles, and gray out orsuppress all other nodes into the background. The common assetsreturned, which may include the control processor where the alarms wereprocessed, the assets shared by these alarms, the I/O modules processingthe alarms, and the like, indicate possible root causes for the alarms.

In some embodiments, users and plant operators can also enter commandsto initiate operations. The HMI may interpret commands written innatural language. For example, to instruct the HMI to initiate fillingup a component labeled “Tank 01,” the command “Fill Tank 01” may beentered into a command bar (not expressly shown).

Returning to FIG. 18, once the ontology is built, then at block 1814,the system may provide an asset namespace registry API in order fordifferent applications to consume the data. Recall from earlier (FIG. 6)that plant assets are organized into an asset registry using namespace.In some embodiments, a RESTful Application Program Interface (e.g., anAzure Active Directory Graph API) may be generated to permit downstreamapplications to extract information from the knowledge graph.

At block 1816, the system provides a way for users to retrieve databased on a process narrative. In some embodiments, the user may retrievethe data using natural-language questions to search for the data. Thesystem may translate the natural-language queries into, for example,SPARQL queries or Gremlin queries. Downstream applications may comprise,for example, an HMI through which commands may be issued from convertedprocess narratives and/or other natural-language commands.

At block 1818, plant control processes and an HMI for the processes maybe designed and developed (i.e., control and HMI engineering) to allow auser to use the system 300 to control the plant and the various plantassets. This involves processing the plant assets and assetrelationships from the asset models to create an HMI asset model. TheHMI asset model arranges the assets in a hierarchical structure (seeFIG. 37), unlike conventional solutions that are discrete in nature(i.e., created based on individual P&ID diagrams). The HMI may then usethe HMI asset model to automatically create a unique, continuous HMIdisplay resembling a two-dimensional plan view at the plant level thatincludes symbols for each asset and lines that represent processes,electrical connections, data links, and the like, extending between thesymbols. Plant operators can then navigate horizontally and verticallyand can zoom in and out on (i.e., along a Z-axis) the two-dimensionalview to move across different assets and verify their behavior.

In some embodiments, the HMI can dynamically change the assets that aredisplayed to the user based on a change in position of the user on theHMI screens and the alarms generated at runtime. The HMI assets shown atruntime can be decided based mainly on two factors: (i) static weightsassigned to the assets based on asset hierarchical details available inthe engineering diagrams (e.g., PFD, P&ID, etc.); and (ii) dynamicweights assigned to each equipment that raises an alarm. Thus, atruntime, the HMI can display a view that includes assets having agreater weight, or if there are alarms, assets that have alarms. The HMIcan also dynamically change the view to include assets for which alarmsare recently raised. Alarms on critical equipment will be given a “high”or “high-high” indicator (e.g., exclamation mark, red color, etc.) andwill get more focus and the complete alarm chain will get highestweightage at runtime. To design the HMI, design specifications for theHMI need to be established.

FIG. 34 shows an exemplary design specification 3400 for an HMIaccording to embodiments of the present disclosure. The specification3400 specifies a hierarchical structure for the HMI that allows the HMIto display certain process control details, including process equipmentdetails and instrumentation control details, in a hierarchicalarrangement. In this example, the HMI is arranged in severalhierarchical display levels 3402, 3404, 3406, 3408. Level 1 displays aprocess area overview, including information such as operational KPIs(key performance indicators), an alarm counter, bypassed signalscounter, trends, overall plant statuses, and the like. The next level,Level 2, displays process unit controls, including controllers, alarms,trends, and statuses. The next level, Level 3, displays process unitdetails, including smaller equipment groups, controllers, alarms,trends, ESD displays, equipment, diagnostics, and statuses. The nextlevel, Level 4, displays process unit support details, includinginterlocks, details, “first outs,” procedures, documentation, and help.Other HMI design specifications may include specifications related thegraphical display, such as alarm priority colors (e.g., red, yellow,orange, etc.) as well as colors indicating status (e.g., out of service,etc.) and function (e.g., feedback, trend lines, etc.). The meanings oficons, symbols, and lines (e.g., solid, dashed, dotted, etc.) may alsobe specified as well as engineering units (e.g., metric, English, etc.),and so forth. These HMI specifications along with the data andinformation extracted from various plant engineering data sources asdiscussed above may then be used to build an HMI asset model.

FIG. 35 illustrates an exemplary HMI asset model 3500 according toembodiments of the present disclosure. The HMI asset model, as the namesuggests, comprises asset models processed and used by the HMI fordisplay to the users. For the present purposes, the HMI asset modelincludes the plant assets and relationships (asset hierarchy),ontological knowledge base, and various plant engineering data sourcesdiscussed previously. The HMI asset model is thus composed of PFDs 3502(e.g., high level equipment, instruments, connections, etc.), PI&Ds andinstrument index 3504 (e.g., equipment, instruments, associationstherebetween, alarms, piping, maintenance override graphics (MOS) andcause and effect (C&E) details, etc.), HMI specification 3506 (e.g.,navigation requirements, alarm aggregation, historization, diagnosticdetails, etc.), control narratives 3508 (e.g., controlling andmonitoring parameters, setpoints, alarms, ranges, engineering units,etc.) as well as knowledge captured (e.g., from maintenance records,previous corrective actions, user experiences, etc.) via engineeringdisplays 3510. This model is automatically updated as the content of thevarious data sources change and assets are added and/or deleted, makingthe model a dynamic HMI asset model.

Once the HMI asset model 3500 (or rather the content thereof) has beenestablished, the model may be used in developing an HMI according toembodiments of the present disclosure. Developing an HMI based on theHMI asset model can involve creating templates 3514 for the variousassets (e.g., instruments, equipment, composites and combinationsthereof, etc.). These templates may include, for example, level symbols(e.g., Level 2, Level 3, etc.), detailed displays of instruments andequipment, instrument attributes and actions, animated scripts, controlengineering templates, consolidated details of pumps, valves, and thelike. Developing the HMI from the HMI asset model can also involvecreating other asset models 3516 in which all areas and units arerepresented as area objects, each composite and/or complex combinationof instruments are instantiated from their templates. This process mayinvolve assigning more weight to some equipment and devices relative toother equipment and devices, with the former being graphicallyemphasized. New equipment and devices introduced, for example, through acontrol narrative, may be added and the asset hierarchy adjustedaccordingly. Equipment and devices may be grouped based on theirassociations in these asset models.

In addition to the above, the HMI asset model can also be used to buildcontrol applications that control operation of the various plant assets.As mentioned earlier, plant assets can be extracted from unstructureddata like engineering diagrams using machine learning to recognize oneor more assets. This procedure can also extract one or more controlloops from the engineering diagrams based on one or more of theequipment, instruments, connectors, and lines to build the dynamic HMIasset model. An auto-control generation procedure may then be used toread one or more control loops from the dynamic asset model and generatecontrol logic for the control applications. The control logic connectsthe control applications with the actual plant equipment to allow thecontrol applications to read or otherwise acquire process values andcontrol operation of the equipment, devices, instruments, and so forth.In other words, the HMI processes the assets and asset relationships andcreates the HMI asset model, arranging the assets in a hierarchicalstructure. This process also links the attributes of each HMI asset witha corresponding control I/O reference so that at runtime, if an operatorturns on a valve in a control application in the HMI, its control valuewill be propagated to the corresponding device in the plant via thecontrol I/O reference (linked in the HMI asset), thus turning on thevalve in the plant.

The HMI developed from the HMI asset model can then be used by a user tomonitor various displays and easily find the root cause of alarm, forexample. In some embodiments, such an HMI may be developed using an HMIdevelopment platform like the System Platform 2017 InTouch OMI(operations management interface) available from Wonderware West ofHouston, Tex. Other HMI development platforms may of course be usedwithin the scope of the present disclosure.

Referring next to FIG. 36, in some embodiments, the HMI according to thepresent disclosure can process alarms raised by various devices andautomatically identify the root causes of the alarms. The HMI candetermine the root cause of an alarm mainly by two factors: (i)process/material flow as defined in an engineering diagram; and (ii)control flow defined in the engineering diagram and any controlnarrative. In the case of a process flow, the HMI first finds the devicenodes causing the different alarms. For each device node, the HMIretrieves the equipment node corresponding to the device node. For thatequipment node, the HMI retrieves any upstream equipment node connectedto that equipment node which also has a device that is raising an alarm.This step is repeated for the neighboring node until the HMI finds aconnected equipment to which no devices are connected that have analarm. This creates a chain of alarms that allows the HMI to determinethe root cause of the alarm by finding the equipment at the highestlevel in the chain that has an alarm.

In the FIG. 36 example, a two-dimensional plan view at the plant levelis displayed by the HMI on a screen 3600 that includes symbols 3602,3604, 3606, and 3608 representing assets and lines 3610, 3612, and 3614extending between the symbols that represent processes, electricalconnections, data links, and the like. Plant operators can then navigatehorizontally and zoom in and out within the two-dimensional view to moveacross different assets and verify their behavior. In this example,material flows from equipment E1215 (pump) to equipment E1214 (boiler)and there is a pressure drop at the inlet of equipment E1214 (boiler).The drop in pressure causes an alarm to be raised by the device(measurement instrument) at the inlet of equipment E1214 (boiler). Theproblem could be with equipment E1215 or with equipment E1209. However,there is no alarm at equipment E1209, so the HMI determines that theproblem is at equipment E1215 (pump). In this example, the throughput ofequipment E1215 (pump) has been reduced and hence there is pressure dropat the connected downstream equipment E1214 (boiler). Thus, although thealarm is raised by the measurement instrument at the boiler E1214, theHMI would identify the root cause of the alarm as the pump E1215 in thisscenario. The HMI assigns additional weight to this equipment fordisplay purposes and places an alarm bulb 3616 next to the equipment.Other alarm bulbs representing other alarms found by the HMI areaggregated at 3618.

Thereafter, the HMI can allow the plant operator to correct the rootcause analysis identified by the system. For example, the operator canmanually select a different equipment and set that equipment as the rootcause for the alarm instead of the system-identified root cause, and canalso manually identify a corresponding alarm chain in some embodiments.These corrections made by the operator can be fed back to the system todynamically update the machine learning algorithms to thereby make thesystem more accurate for subsequent alarm root cause analysis.

Referring now to FIG. 37, in some embodiments, an HMI according to thepresent the disclosure may aggregate the alarms in a plant and provide adisplay screen 3700 showing a plant level view of the aggregated alarms.In some embodiments, the system may aggregate alarms by obtaining a listof all equipment generating alarms and identifying the first equipmentfrom the list. The system adds this equipment to a new alarm cluster andchecks whether a neighboring equipment is also generating alarms. Ifyes, then the system adds the neighboring equipment to the current alarmcluster. The process is repeated with the next equipment on the listuntil all equipment and neighboring equipment have been processed. Theplant level aggregated alarms are depicted as a series of bulbs 3702 ina designated area of the screen 3700 (e.g., upper left corner) in theexample shown, although a different location or different icons orgraphical symbol may certainly be used for the alarms. Each of the bulbs3702 corresponds to a respective component (e.g., device, equipment,etc.) for which an alarm was raised in the plant. As can be seen, thereare five components 3704 a, b, c, d, e for which an alarm was raised,each component bearing a bulb icon 3706 to indicate an alarm conditionat that component.

Selecting (e.g., by tapping, double-clicking, etc.) one of thecomponents 3704 a-e causes the HMI to display information about thatcomponent, including which downstream components may be potential rootcauses for the alarm. In some embodiments, the system may determinepotential root causes for an alarm by identifying the alarm at thehighest equipment level, then finding a source equipment connected tothis equipment that also has alarm. The system then drills down on thesource equipment to find a connected subsequent equipment that also hasalarm. The process is repeated until the lowest connected equipment thathas an alarm is found.

FIG. 38 graphically illustrates an embodiment of an HMI that candynamically zoom in (and out) on assets according to the position of auser/observer on an asset hierarchy screen. The position of the observeron the hierarchy screen is determined mainly by two factors: (1) theposition on the ground as determined by the screen coordinates (e.g., xand y coordinates), and (2) the height of the observer from the ground.Based on the position of the observer on the screen, relevantinformation around the asset model is provided for viewing. Consider,for example, when the user is positioned at Area 1 with respect to theHMI screen and there are alarm conditions at Equipment 5, Equipment 3,Device 2, and Device 4. In this example, the HMI screen would showEquipment 7, 1, 2, 4, 5, and 6 and any relevant information about theequipment, but not Equipment 8, 9, or 3, or any of the devices connectedto Equipment 3 yet. Assume the user now zooms in so that the position ofthe user is at Unit 3. In that example, the HMI screen would showEquipment 7, 1, 2, 4, 5, and 6 as before, but would also show Equipment8, 9, or 3. Once the user moves further down to Equipment 5, then theHMI screen would show Devices 1, 2, 3, 4, and 5 and any relevantinformation about these devices. The HMI screen automatically removesany equipment that are above the level of the user.

FIG. 39 illustrates an exemplary HMI display screen 3900 that candynamically zoom in (and out) on assets according to the position of auser/observer according to embodiments of the present disclosure. Basedon the zoom level as determined by the current position of the user andthe current active alarms in the plant, the dynamic HMI display screen3900 shows relevant information around the asset model. In the exampleon the left of FIG. 39, the HMI display screen 3900 shows theconnectivity between assets 3902 and 3904. In the middle example, theuser has zoomed in, so the HMI display screen 3900 remaps theconnectivity to show an additional asset 3906 on the screen. In theexample on the right, the user has zoomed in further, so the HMI displayscreen 3900 remaps the connectivity again to show yet an additionalasset 3908 on the screen.

Referring now to FIGS. 40A and 40B, an HMI according to embodiments ofthe present disclosure can also perform alarm aggregation at an areaLevel, in addition to the plant level alarm aggregation discussed above.FIG. 40A graphically illustrates an exemplary embodiment of such an HMIin which the HMI uses a 2-dimensional view of the plant assets to showarea level alarm aggregation. Starting with the example on the left,solid lines such as line 4002 indicate horizontal connectivity within aplane and dashed lines such as line 4004 represent vertical connectivityacross multiple planes (i.e., a 2-dimensional view). As can be seen,area level assets A1 and A2 have alarm conditions that are indicated bythe high and critical alarm icons, respectively. When the user selectsA2 (e.g., by tapping, clicking, etc.), the HMI takes the user to theexample on the right, where the user finds a critical alarm at asset U3and a high alarm at asset U2.

If the user selects U2, the HMI takes the user to the example on theleft in FIG. 40B, where the user finds a critical alarm at equipmentU3E2. From here, the user will be able to visually view the source ofthe critical alarm at U2E1, which cascaded through U3E1 and also U3E2.Assuming the user selects U2E1, the HMI takes the user to the example onthe right in FIG. 40B, where the user will be shown the device levelassets, including the source of the problem that cause the alarms, D1.Selecting D1 will provide the user with information about that device,any trends seen at the device, any support and guidance available forthe device, and any knowledge captured about the device.

Accordingly, as described herein, embodiments of the present disclosureprovide systems and methods for controlling industrial processautomation and control systems. The methods and systems automatically,and through the use of machine learning (ML) models and algorithms,extract plant assets from engineering diagrams and other plantengineering data sources, establish asset relationships to create aplant asset registry and build an asset hierarchy from the plant assets,generate an ontological knowledge base from the plant asset hierarchy,and provide an HMI for controlling the industrial process based on theplant asset hierarchy and the ontological knowledge base.

Such embodiments of the present disclosure may comprise a specialpurpose computer including a variety of computer hardware, as describedin greater detail below. Embodiments within the scope of the presentdisclosure also include computer-readable media for carrying or havingcomputer-executable instructions or data structures stored thereon. Suchcomputer-readable media can be any available media that can be accessedby a special purpose computer and comprises computer storage media andcommunication media. By way of example, and not limitation, computerstorage media include both volatile and nonvolatile, removable andnon-removable media implemented in any method or technology for storageof information such as computer-readable instructions, data structures,program modules or other data. Computer storage media are non-transitoryand include, but are not limited to, random access memory (RAM), readonly memory (ROM), electrically erasable programmable ROM (EEPROM),compact disk ROM (CD-ROM), digital versatile disks (DVD), or otheroptical disk storage, solid state drives (SSDs), magnetic cassettes,magnetic tape, magnetic disk storage, or other magnetic storage devices,or any other medium that can be used to carry or store desirednon-transitory information in the form of computer-executableinstructions or data structures and that can be accessed by a computer.When information is transferred or provided over a network or anothercommunications connection (either hardwired, wireless, or a combinationof hardwired or wireless) to a computer, the computer properly views theconnection as a computer-readable medium. Thus, any such connection isproperly termed a computer-readable medium. Combinations of the aboveshould also be included within the scope of computer-readable media.Computer-executable instructions comprise, for example, instructions anddata which cause a general-purpose computer, special purpose computer,or special purpose processing device to perform a certain function orgroup of functions.

The following discussion is intended to provide a brief, generaldescription of a suitable computing environment in which aspects of thedisclosure may be implemented. Although not required, aspects of thedisclosure will be described in the general context ofcomputer-executable instructions, such as program modules, beingexecuted by computers in network environments. Generally, programmodules include routines, programs, objects, components, datastructures, etc. that perform particular tasks or implement particularabstract data types. Computer-executable instructions, associated datastructures, and program modules represent examples of the program codemeans for executing steps of the methods disclosed herein. Theparticular sequence of such executable instructions or associated datastructures represent examples of corresponding acts for implementing thefunctions described in such steps.

Those skilled in the art will appreciate that aspects of the disclosuremay be practiced in network computing environments with many types ofcomputer system configurations, including personal computers, hand-helddevices, multi-processor systems, microprocessor-based or programmableconsumer electronics, network PCs, minicomputers, mainframe computers,and the like. Aspects of the disclosure may also be practiced indistributed computing environments where tasks are performed by localand remote processing devices that are linked (either by hardwiredlinks, wireless links, or by a combination of hardwired or wirelesslinks) through a communications network. In a distributed computingenvironment, program modules may be located in both local and remotememory storage devices.

An exemplary system for implementing aspects of the disclosure includesa special purpose computing device in the form of a conventionalcomputer, including a processing unit, a system memory, and a system busthat couples various system components including the system memory tothe processing unit. The system bus may be any of several types of busstructures including a memory bus or memory controller, a peripheralbus, and a local bus using any of a variety of bus architectures. Thesystem memory includes computer storage media, including nonvolatile andvolatile memory types. A basic input/output system (BIOS), containingthe basic routines that help transfer information between elementswithin the computer, such as during start-up, may be stored in ROM.Further, the computer may include any device (e.g., computer, laptop,tablet, PDA, cell phone, mobile phone, a smart television, and the like)that is capable of receiving or transmitting an IP address wirelessly toor from the internet.

The computer may also include a magnetic hard disk drive for readingfrom and writing to a magnetic hard disk, a magnetic disk drive forreading from or writing to a removable magnetic disk, and an opticaldisk drive for reading from or writing to removable optical disk such asa CD-ROM or other optical media. The magnetic hard disk drive, magneticdisk drive, and optical disk drive are connected to the system bus by ahard disk drive interface, a magnetic disk drive-interface, and anoptical drive interface, respectively. The drives and their associatedcomputer-readable media provide nonvolatile storage ofcomputer-executable instructions, data structures, program modules, andother data for the computer. Although the exemplary environmentdescribed herein employs a magnetic hard disk, a removable magneticdisk, and a removable optical disk, other types of computer readablemedia for storing data can be used, including magnetic cassettes, flashmemory cards, digital video disks, Bernoulli cartridges, RAMs, ROMs,SSDs, and the like.

Communication media typically embody computer readable instructions,data structures, program modules or other data in a modulated datasignal such as a carrier wave or other transport mechanism and includesany information delivery media.

Program code means comprising one or more program modules may be storedon the hard disk, magnetic disk, optical disk, ROM, and/or RAM,including an operating system, one or more application programs, otherprogram modules, and program data. A user may enter commands andinformation into the computer through a keyboard, pointing device, orother input device, such as a microphone, joy stick, game pad, satellitedish, scanner, or the like. These and other input devices are oftenconnected to the processing unit through a serial port interface coupledto the system bus. Alternatively, the input devices may be connected byother interfaces, such as a parallel port, a game port, or a universalserial bus (USB). A monitor or another display device is also connectedto the system bus via an interface, such as a video adapter. In additionto the monitor, personal computers typically include other peripheraloutput devices (not shown), such as speakers and printers.

One or more aspects of the disclosure may be embodied incomputer-executable instructions (i.e., software), routines, orfunctions stored in system memory or nonvolatile memory as applicationprograms, program modules, and/or program data. The software mayalternatively be stored remotely, such as on a remote computer withremote application programs. Generally, program modules includeroutines, programs, objects, components, data structures, etc. thatperform particular tasks or implement particular abstract data typeswhen executed by a processor in a computer or other device. The computerexecutable instructions may be stored on one or more tangible,non-transitory computer readable media (e.g., hard disk, optical disk,removable storage media, solid state memory, RAM, etc.) and executed byone or more processors or other devices. As will be appreciated by oneof skill in the art, the functionality of the program modules may becombined or distributed as desired in various embodiments. In addition,the functionality may be embodied in whole or in part in firmware orhardware equivalents such as integrated circuits, application specificintegrated circuits, field programmable gate arrays (FPGA), and thelike.

The computer may operate in a networked environment using logicalconnections to one or more remote computers. The remote computers mayeach be another personal computer, a tablet, a PDA, a server, a router,a network PC, a peer device, or other common network node, and typicallyinclude many or all of the elements described above relative to thecomputer. The logical connections include a local area network (LAN) anda wide area network (WAN) that are presented here by way of example andnot limitation. Such networking environments are commonplace inoffice-wide or enterprise-wide computer networks, intranets and theInternet.

When used in a LAN networking environment, the computer is connected tothe local network through a network interface or adapter. When used in aWAN networking environment, the computer may include a modem, a wirelesslink, or other means for establishing communications over the wide areanetwork, such as the Internet. The modem, which may be internal orexternal, is connected to the system bus via the serial port interface.In a networked environment, program modules depicted relative to thecomputer, or portions thereof, may be stored in the remote memorystorage device. It will be appreciated that the network connectionsshown are exemplary and other means of establishing communications overwide area network may be used.

Preferably, computer-executable instructions are stored in a memory,such as the hard disk drive, and executed by the computer.Advantageously, the computer processor has the capability to perform alloperations (e.g., execute computer-executable instructions) inreal-time.

The order of execution or performance of the operations in embodimentsof the disclosure illustrated and described herein is not essential,unless otherwise specified. That is, the operations may be performed inany order, unless otherwise specified, and embodiments of the disclosuremay include additional or fewer operations than those disclosed herein.For example, it is contemplated that executing or performing aparticular operation before, contemporaneously with, or after anotheroperation is within the scope of aspects of the disclosure.

Embodiments of the disclosure may be implemented withcomputer-executable instructions. The computer-executable instructionsmay be organized into one or more computer-executable components ormodules. Aspects of the disclosure may be implemented with any numberand organization of such components or modules. For example, aspects ofthe disclosure are not limited to the specific computer-executableinstructions or the specific components or modules illustrated in thefigures and described herein. Other embodiments of the disclosure mayinclude different computer-cutable instructions or components havingmore or less functionality than illustrated and described herein.

When introducing elements of aspects of the disclosure or theembodiments thereof, the articles “a”, “an”, “the” and “said” areintended to mean that there are one or more of the elements. The terms“comprising”, “including”, and “having” are intended to be inclusive andmean that there may be additional elements other than the listedelements.

Having described aspects of the disclosure in detail, it will beapparent that modifications and variations are possible withoutdeparting from the scope of aspects of the disclosure as defined in theappended claims. As various changes could be made in the aboveconstructions, products, and methods without departing from the scope ofaspects of the disclosure, it is intended that all matter contained inthe above description and shown in the accompanying drawings shall beinterpreted as illustrative and not in a limiting sense.

1. A control system for an industrial plant, comprising: one or moreprocessors; and a storage unit communicatively coupled to the one ormore processors and storing processor-executable instructions thereonthat, when executed by the one or more processors, cause the controlsystem to: perform a process that inputs an engineering diagram for aunit of the industrial plant, the engineering diagram including symbolsrepresenting assets of the industrial plant; perform a process thatextracts one or more assets from the engineering diagram using machinelearning to recognize the one or more assets, the one or more assetsincluding equipment, instruments, connectors, and lines, the linesrelating the equipment, instruments, and connectors to one another;perform a process that determines one or more relationships between theequipment, instruments, connectors, and lines to one another usingmachine learning to recognize the one or more relationships; and performa process that creates a flow graph from the equipment, instruments,connectors, and lines and the relationships between the equipment,instruments, connectors, and lines.
 2. The control system of claim 1,wherein the process that extracts one or more assets from theengineering diagram includes a geometry-based extraction process and amachine learning-based classification process.
 3. The control system ofclaim 1, wherein the processor-executable instructions further cause thecontrol system to perform a process that extracts one or more controlloops from the engineering diagram based on one or more of theequipment, instruments, connectors, and lines.
 4. The control system ofclaim 1, wherein the processor-executable instructions further cause thecontrol system to perform a process that extracts a unit identifier fromthe engineering diagram, wherein the unit identifier uniquely identifiesa particular unit from among multiple units in the industrial plant. 5.The control system of claim 1, wherein the processor-executableinstructions further cause the control system to perform a process thatextracts a drawing number and a revision number from the engineeringdiagram, the drawing number uniquely identifying the engineering diagramfrom among multiple engineering diagrams for the industrial plant andthe revision number indicating a revision of the engineering diagramwhich is incremented whenever there are changes in the engineeringdiagram.
 6. The control system of claim 4, wherein theprocessor-executable instructions further cause the control system toperform a process that assigns a unique identifier to each equipment,instrument, connector, and line of the engineering diagram using theunit identifier from the engineering diagram.
 7. The control system ofclaim 1, wherein the process that determines one or more relationshipsbetween the equipment, instruments, connectors, and lines includes aprocess that generates a line-line graph, a process that generates anequipment-line graph, a process that generates a connector-line graph,and a process that generates an instrument-line graph.
 8. The controlsystem of claim 1, wherein the processor-executable instructions furthercause the control system to create the flow graph for the engineeringdiagram by performing a process that merges the line-line graph, theequipment-line graph, the connector-line graph, and the instrument-linegraph with one another.
 9. The control system of claim 7, wherein theprocessor-executable instructions further cause the control system toperform a process that merges flow graphs from multiple engineeringdiagrams for the industrial plant to create a flow graph for theindustrial plant.
 10. The control system of claim 1, wherein theprocessor-executable instructions further cause the control system toperform a process that displays the equipment, instruments, connectorsand lines and the relationships between the equipment, instruments,connectors and lines in an asset hierarchy.
 11. A method of controllingan industrial plant, comprising: inputting an engineering diagram for aunit of the industrial plant, the engineering diagram including symbolsrepresenting assets of the industrial plant; extracting one or moreassets from the engineering diagram using machine learning to recognizethe one or more assets, the one or more assets including equipment,instruments, connectors, and lines, the lines relating the equipment,instruments, and connectors to one another; determining one or morerelationships between the equipment, instruments, connectors, and linesto one another using machine learning to recognize the one or morerelationships; and creating a flow graph from the equipment,instruments, connectors, and lines and the relationships between theequipment, instruments, connectors, and lines.
 12. The method of claim11, wherein extracting one or more assets from the engineering diagramincludes a performing geometry-based extraction process and a machinelearning-based classification process.
 13. The method of claim 11,further comprising extracting one or more control loops from theengineering diagram based on one or more of the equipment, instruments,connectors, and lines.
 14. The method of claim 11, further comprisingextracting a unit identifier from the engineering diagram, wherein theunit identifier uniquely identifies a particular unit from amongmultiple units in the industrial plant.
 15. The method of claim 11,further comprising extracting a drawing number and a revision numberfrom the engineering diagram, the drawing number uniquely identifyingthe engineering diagram from among multiple engineering diagrams for theindustrial plant and the revision number indicating a revision of theengineering diagram which is incremented whenever there are changes inthe engineering diagram.
 16. The method of claim 14, further comprisingassigning a unique identifier to each equipment, instrument, connector,and line of the engineering diagram using the unit identifier from theengineering diagram.
 17. The method of claim 11, wherein determining oneor more relationships between the equipment, instruments, connectors,and lines includes generating a line-line graph, an equipment-linegraph, a connector-line graph, and an instrument-line graph.
 18. Themethod of claim 11, wherein creating the flow graph for the engineeringdiagram includes merging the line-line graph, the equipment-line graph,the connector-line graph, and the instrument-line graph with oneanother.
 19. The method of claim 17, further comprising merging flowgraphs from multiple engineering diagrams for the industrial plant tocreate a flow graph for the industrial plant.
 20. The method of claim11, further comprising displaying the equipment, instruments, connectorsand lines and the relationships between the equipment, instruments,connectors and lines in an asset hierarchy.
 21. (canceled)